Unpaired Image Captioning by Image-level Weakly-Supervised Visual Concept Recognition

The goal of unpaired image captioning (UIC) is to describe images without using image-caption pairs in the training phase. Although challenging, we except task can be accomplished by leveraging a set aligned with visual concepts. Most existing studies use off-the-shelf algorithms obtain concepts because Bounding Box (BBox) labels or relationship-triplet used for are expensive acquire. In order resolve problem annotations, propose novel approach achieve cost-effective UIC. Specifically, adopt image-level optimization UIC model weakly-supervised manner. For each image, assume that only (such as object categories and relationships) available specific locations numbers. utilized train recognition extract information (e.g., instance) an extracted instances adopted infer relationships among different objects based on enhanced graph neural network (GNN). proposed achieves comparable even better performance compared previous methods cost annotations. Furthermore, design unrecognized (UnO) loss combined concept reward improve alignment inferred relationship images. It effectively alleviate issue encountered models about generating sentences nonexistent objects. To best our knowledge, this first attempt solve Weakly-Supervised (WS-UIC) labels. Extensive experiments have been carried out demonstrate WS-UIC inspiring results COCO dataset while significantly reducing labeling.